MEMORANDUM May 4, 2016 


TO: Lance Menster 
Officer of Elementary Curriculum and Development 


Annie Wolfe 
Officer of Secondary Curriculum and Development 


FROM: Carla Stevens 
Assistant Superintendent, Research and Accountability 


SUBJECT: | THINK THROUGH MATH IN HISD, 2014-2015 


Think Through Math (TTM) is an online, adaptive mathematics program designed to supplement 
classroom instruction. In 2014—2015, the program was funded by the Texas Education Agency 
for students in grades 3-8, and schools had the option to purchase the program for students in 
other grades. This report compares the STAAR mathematics achievement of HISD students in 
grades 3-8 who enrolled in the TTM program in 2014—2015 with the results of similar students 
in the same grades who did not use TTM. Comparisons were made for students districtwide, 
for students based on 2014 STAAR Mathematics performance level, and for students who 
attended a 2014 Improvement Required (IR) school. 


Key findings include: 

e A total of 43,997 HISD students used TTM in 2014—2015. Nearly all the students, 99 
percent, were enrolled in grades 3-8, the grade levels for which the program was supported 
by TEA. 

e Students enrolled in TTM completed an average of 14 TTM lessons and passed an average 
of four on-grade level lessons in the academic year. 

e Propensity score matching, which matched TTM users with non-TTM users based on key 
variables, indicated no significant difference in 2015 STAAR Mathematics scale scores 
between TTM users and non-TTM users. 

e Propensity score matching was also used to confirm that there were no significant 
differences between TTM users and non-TTM users based on 2014 STAAR Mathematics 
performance or enrollment in an IR school. Specifically, there were no significant 
differences in 2015 STAAR Mathematics scale scores of matched TTM and non-TTM users 
who did not meet the 2014 STAAR Mathematics standard, who met but did not achieve the 
advanced 2014 STAAR Mathematics standard, who met the 2014 STAAR Mathematics 
advanced standard, or who attended a 2014 IR school. 


Further distribution of this report is at your discretion. Should you have any further questions, 
please contact me at 713-556-6700. 
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THINK THROUGH MATH IN HISD, 2014-2015 


Executive Summary 


Evaluation Description 

Think Through Math (TTM) is an online, adaptive mathematics program designed to supplement 
classroom instruction in mathematics. Beginning in August 2012, the program was made available free of 
charge to students in grades three through eight by the Texas SUCCESS (Students Using Curriculum 
Content to Ensure Sustained Success) initiative through the Texas Education Agency (TEA). Additional 
services could be purchased by campuses to serve students through high school Algebra | and geometry. 
This study was designed to investigate the impact of the Think Through Math program on the State of 
Texas Assessments of Academic Readiness (STAAR) mathematics achievement of HISD students in 
grades three through eight who used the program during the 2014—2015 academic year. 


Highlights 


A total of 43,997 HISD students used TTM in 2014-2015. Nearly all the students, 99 percent, were 
enrolled in grades 3-8, the grade levels for which the program was supported by TEA. 


Students enrolled in TTM completed an average of 14 TTM lessons and passed an average of four 
on-grade level lessons in the academic year. 


Of the students enrolled in TTM in 2014—2015, 28,412, 65 percent, earned scores on both the 2014 
and 2015 spring STAAR Mathematics exams and were included in analyses of the program. An 
additional 28,959 HISD students earned scores on the same exams and did not use TTM and were 
also used in the analyses. 


Propensity score matching, which matched TTM users with non-TTM users based on key variables 
such as 2014 STAAR Mathematics scale scores, demographics, special education and gifted/talented 
status, and enrollment in a 2014 Improvement Required (IR) school, indicated no significant 
difference in 2015 STAAR Mathematics scale scores between TTM users and non-TTM users. 


The majority of students in the analyses achieved the same standard on the 2015 STAAR 
mathematics exam that they had achieved on the 2014 STAAR Mathematics exam, at about the 
same rates among both TTM and non-TTM users. These results were achieved irrespective of the 
numbers of TTM lessons completed or the number of TTM on-grade level lessons passed by TTM 
users. 


Propensity score matching was also used to confirm that there were no significant differences 
between TTM users and non-TTM users based on 2014 STAAR Mathematics performance or 
enrollment in a 2014 IR school. Specifically, there were no significant differences in 2015 STAAR 
Mathematics scale scores of matched TTM and non-TTM users who did not meet the 2014 STAAR 
Mathematics standard, who met but did not achieve the advanced 2014 STAAR Mathematics 
standard, who met the 2014 STAAR Mathematics advanced standard, or who attended a 2014 IR 
school. 
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Recommendations 

e TTM data would be more accessible and reliable if it were associated with a single unique identifying 
number, such as the Texas Student Data System (TSDS) unique ID or the HISD local student ID, 
rather than based on a combination of user names and identifying numbers provided by another 
server (Clever). Further, it would be helpful if TTM supported a system that allowed data to be 
modified without loss of information, and if data were accessible for more than a single academic 
year. It is recommended that TTM be encouraged to support a system that allows linking to a single 
student identification number per user and access to records for each individual student for multiple 
years. 


e Currently, the TTM system does not allow easy access to the number of discrete lessons a student 
uses. For example, of two students who complete 15 lessons, one could have completed 15 discrete 
lessons and the other could have completed a single lesson 15 times. To allow better measurement 
and understanding of students’ academic progress in the program, it is recommended that TTM be 
encouraged to routinely collect data on both the number of discrete lessons and the number of 
discrete on-grade-level lessons a student attempts, completes, and passes. 


e In 2014—2015, no significant differences were found in the STAAR Mathematics performance of TTM 
users compared with the performance of non-TTM users, but in 2013-2014, TTM was shown to be an 
effective academic support for HISD students in increasing their scale scores on the STAAR 
Mathematics assessment. Because 2014-2015 issues in data management may have contributed to 
the contradictory results, it is recommended that the program continue being made accessible to 
schools and teachers as a supplement to regular instruction in order to allow further evaluation of the 
effectiveness of the program using a complete data set. 


e Further, it is recommended that schools and teachers who choose to use the program strive to 
monitor and increase students’ use of the program regularly and with the purpose of achieving 
specific mathematics objectives. 


Administrative Response 
e TTMwill continue to be provide free through the 2016-2017 school year by the state of Texas 


(see http://tea.texas.gov/interiorpage_wide.aspx?id=25769824546). 


e Significant to this data is the shift in implementation and assessment of new grades 3-8 
mathematics standards during the 2014-2015 school year. 


e TTM is accessible to students in grades 3-8, as well as their math teachers, through the HISD 
HUB’s digital resources page. 


e The program is intended for use by all students in grades 3-8, and it is recommended that 
students spend approximately 75 minutes or more using TTM each week. This equates to 


approximately 1 to 2 lessons per week. 


e Previous research (from 2013-2014) has shown that when students “Think 30”—that is, complete 
and pass 30 lessons before STAAR, their scores are positively impacted. 
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e —Inlight of the rigorous new mathematics standards that were introduced in grades 3-8 during the 
2014-2015 school year, the district has been working towards a more consistent and 
comprehensive plan for implementing TTM strategically as the primary Response to Intervention 
(RTI) program. It is thus recommended, that campuses and teachers utilize the program with 
fidelity. 


e = In addition—and in light of the new Algebra | standards introduced during the 2015-2016 school 
year—the state of Texas is now also providing TTM for free for all Algebra | students, including 
re-testers. 


e Elementary and Secondary Curriculum and Development will continue to provide information to 


lead teachers and department chairs on TTM trainings, specialized summer school TTM 
pathways, and free TTM-hosted webinars. 
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Introduction 


Think Through Math (TTM) is an adaptive, web-based mathematics enrichment program funded through 
the Texas SUCCESS program provided by the Texas Education Agency (TEA) for students in grades 
three through eight. Since the lessons are online, students can access the program both at school and 
during after-school hours, free of charge. The program is also available for campuses to purchase for 
students in high school who are studying mathematics through Algebra | and geometry. 


Typically, a student takes a placement test at the beginning of a school year and, based on the results, is 
placed on a grade-level “pathway” which consists of approximately 40 target lessons. Additional 
remediation lessons, are inserted on a student’s pathway as needed, and students can begin a new 
pathway when they finish the grade level assigned. Teachers have significant control on the presentation 
of the curriculum and can reorder, add, and remove lessons from a student's pathway to meet 
instructional and/or student needs. The curriculum is designed to supplement rather than replace 
classroom teaching. 


A notable element of the program is the student motivators built into the lessons. Students earn points for 
all the actions they take in the program. Points can be used in a variety of ways, including to contribute to 
a class reward such as a pizza party or classroom supplies, to make a cash contribution to the student’s 
choice of a variety of charities, or to enhance a student’s online TTM avatar. Students can earn points 
even for answering questions randomly, so teachers’ monitoring of use of the program is crucial. 
Teachers have ready access to information about each student’s progress and can instantly change a 
student’s pathway or send a personal note to a student. 


Research completed in 2013-2014 for the state of Texas and for HISD indicated that TTM is an effective 
support for increasing student performance on the STAAR Mathematics exam. According to the Gibson 
Consulting Group, reporting on a study for the TEA: 


Advanced statistical modeling, accounting for other observable factors that may influence student 
outcomes—such as students’ prior STAAR-Mathematics performance—showed that students in 
Grades 3-8 who attempted 20 or more Think Through Math lessons had statistically significant 
higher STAAR-Mathematics scores than non-users (Garland, et al, n.d., p. 12). 


In HISD, 2013-2014 students who used TTM were matched with HISD students who did not use the 
program based on key factors such as previous STAAR Mathematics scale scores, state school rating, 
and demographics. Results showed that 2013-2014 HISD students who used TTM achieved significantly 
higher 2014 STAAR Mathematics scale scores than did similar students who did not. 


This report provides demographic and TTM achievement information for HISD students who used the 
program in 2014—2015 and compares the STAAR 3-8 Mathematics performance of TTM users with the 
results of similar HISD students who did not use the program. All students who took both the spring 2014 
and spring 2015 STAAR Mathematics assessment are included in the analyses. To determine if the TTM 
program was particularly effective with students with focused needs such as enrichment and remediation, 
the same analyses were performed for subgroups of students disaggregated by students’ spring 2014 
STAAR Mathematics performance and attendance at a school with the 2014 TEA accountability rating 
Improvement Required (IR). 
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Methods 


Data Collection and Analysis 


e =Think Through Math achievement data for student work completed by April 17, 2015 came from TTM. 
Students who were identified as having started the program and students who completed the initial 
TTM placement test were included as TTM users. 


e TTM achievement is reported as the number of lessons TTM users completed and the number of on- 
grade level lessons they passed. Completed lessons were those that students finished but did not 
necessarily pass. Completed lessons included target lessons, specifically aligned to the grade level 
curriculum, and remedial or administrative lessons, inserted to allow students to negotiate the 
program and to provide lessons on content students had not yet mastered. On-grade level lessons 
passed were those lessons specifically designed to address grade level content on which students 
were successful in the summative assessment. Data for each category were provided by TTM. 


e Demographic data were drawn from PEIMS Fall 2014 Resubmission files. Limited English Proficient 
(LEP) students included those identified as LEP within the last two years (codes 1, F, and S). 


e Improvement Required (IR) schools were those identified by the Texas Education Agency in the 2014 
Accountability Ratings. School ratings, which include Met Standard, Met Alternative Standard, and 
Improvement Required, are based on results on four indices, which include a range of academic 
performance indicators. The IR rating is defined by the state as, “Improvement Required. 
Unacceptable rating assigned to districts, campuses, charter operators, and alternative education 
campuses (AECs) that miss the target on one or more performance indexes” (TEA, 2014, page 13). 


e Data on STAAR performance came from data files from the Texas Education Agency. Scored 
versions of the 2013-2014 and 2014—2015 first administrations of the regular STAAR, both English 
and Spanish, and STAAR L (the linguistically accommodated test for English language learners) were 
used for the analyses. Students who had earned scores on both the spring 2014 STAAR 
Mathematics and spring 2015 Mathematics exams, both those who had used TTM in 2014-2015 
(28,412 students) and those who had not (28,959 students) were included in the analyses. 


e Propensity score nearest neighbor matching was used to compare the performance of students who 
took both the spring 2014 and 2015 STAAR Mathematics assessments and used TTM in 2014-2015 
with the performance of other students in the district who took the same tests. Propensity score 
matching was designed to allow causal inferences based on observational data (Cohen, 1988). 
Students in grades three through eight were matched on spring 2014 STAAR Mathematics scale 
score, 2014-2015 enrollment in a 2014 Improvement Required (IR) school, grade level, gender, 
race/ethnicity, limited English proficiency (LEP) status, economic disadvantage, special education 
status, and gifted/talented status. For each comparison, a probit regression analysis indicated 
significant differences between the groups initially. All students in each comparison group were 
included in the matching procedure and, following the matching analysis, none of the bias reached 
the five percent level, indicating no significant differences between the matched groups. 
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e In this report, numbers were rounded to the nearest whole number in the text, and to the nearest 
tenth in the tables. Numbers were rounded up if the next digit was five or higher and were not 
changed if the next digit was lower, so 11.49 was recorded as 11.5 in a table and 11 in the text, while 
11.50 was recorded as 11.5 in the table and 12 in the text. 


e For percentages associated with demographics, percentages in the text are reported as percentage 
of TTM users or non-TTM users who share a given characteristic. In the tables, percentages are 
reported as a distribution of a given characteristic within the group of TTM users or the group of non- 
TTM users. Percentages reported in the text can be reproduced by using the corresponding table of 
demographics, selecting a single demographic characteristic, and dividing the number of students in 
the chosen group, TTM user or non-TTM user, by the sum of the number of TTM users and non-TTM 
users who shared the same characteristic. 


Data Limitations 

Of the total number of TTM users, 1,757 TTM students (four percent) used multiple TTM accounts, 
sometimes associated with different ID numbers and/or different schools. For analyses, identifiable 
students were matched with their local ID numbers and the numbers of TTM lessons were summed and 
associated with the first school assigned by TTM, resulting in some lessons being credited to schools at 
which they were not completed. 


Due to technical issues in loading student information from HISD to TTM at the beginning of the 2014— 
2015 school year, some student TTM achievement data was lost. Students who began working with TTM 
materials before the correct upload was finalized were credited with incorrect totals of discrete lessons 
they completed. For example, according to the TTM file, 1,875 students completed between one and 112 
lessons, but zero discrete lessons. TTM does not regularly report on numbers of discrete lessons that 
students complete so resolving the issue would have taken more time than was available at the end of 
the academic year. In order to document the achievement of as many 2014—2015 TTM users as possible 
while avoiding including inaccurate data, only TTM lessons completed and TTM on-grade-level lessons 
passed are reported in this evaluation. To further clarify, lessons completed and on-grade-level lessons 
passed are not discrete lessons; i.e., students could complete and pass the same lesson multiple times. 
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Results 
How many HISD students used TTM in 2014-2015 and what were their demographics? 


e A total of 43,997 HISD students were identified as TTM users and were matched with demographic 
data in 2014-2015. In 2013-2014, 51,863 HISD students met the same criteria (Department of 
Research and Accountability, 2014), a 15 percent decrease in the number of participants between the 
years. 


e Shown in Table 1 (page 19) and illustrated in Figure 1, though students at every grade level 
participated in TTM both years, in 2014—2015, 99 percent of TTM participants were in grades 3-8, the 
grades targeted by the program, while in 2013-2014, 95 percent of TTM students were in the same 
grades. 


Figure 1. Percentage of TTM students at each grade level, 2013-2014 and 2014-2015 
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Note: Percentages may not equal 100 due to rounding. 
Sources: PEIMS Fall Resubmission and TTM files, 2013-2014 and 2014-2015 


e The majority of TTM users were in elementary school grades three through five both years, 64 
percent in 2014—2015 and 68 percent in 2013-2014. Nearly 36 percent were in middle school in 
2014—2015 and 27 percent were in middle school in 2013-2014. 


e =©As detailed in Table 1 (page 19), 13 percent of TTM users attended 2014 Improvement Required (IR) 
schools, while in 2013-2014, 14 percent of TTM participants attended the same schools. 
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HISD Research and Accountability 


The demographics of TTM users in 2014—2015 largely paralleled those of 2013-2014 TTM users, 
with the largest exception being the percentage of economically disadvantaged students. In 2013- 
2014, 83 percent of HISD TTM users were economically disadvantaged while 76 percent were 
economically disadvantaged in 2014—2015. More detail about demographics of TTM users can be 
found in Table 1 (page 19). 


2014-2015 TTM achievement, documented by number of lessons completed and number of on- 
grade level lessons completed at each grade level, is illustrated in Figure 2 and detailed in Table 2 
(page 20). Students in HISD completed an average of 14 TTM lessons (on grade-level and remedial, 
including repetitions of lessons) in 2014-2015, with the highest average numbers earned in 
kindergarten and grade two, and the lowest average number in grade nine. 


Though TTM provided approximately 40 on-grade level lessons for each grade level, students in 
HISD passed only an average of four on-grade level lessons (including repetitions of lessons). The 
highest average number of on-grade level lessons passed was six, for students in grade three. 


Figure 2. Average number of TTM lessons completed and average number of on-grade level 
TTM lessons passed by grade level, 2014-2015 


Mean Number 


m™Lessons Completed m On-Grade Lessons Passed 


Sources: PEIMS Fall Resubmission and TTM files, 2014-2015 


e Of the 43,997 HISD students who used TTM in 2014-2015, 28,412 took both the spring 2014 and 
2015 STAAR Mathematics assessments. Another 28,959 HISD students took the same exams at the 
same times but did not use TTM in 2014—2015. Shown in Figure 3, about half the students who took 
both exams used TTM and about half did not. For comparison, in 2013-2014, 53 percent of students 
who took both the 2013 and 2014 STAAR exams used TTM and 47 percent did not. 


Figure 3. Percentage of TTM users and non-TTM users among 2013-2014 students who took 
both the 2013 and 2014 STAAR Mathematics assessments, and among 2014-2015 students 
who took both the 2014 and 2015 STAAR Mathematics assessments 
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Sources: PEIMS Fall Resubmission and TTM files, 2013-2014 and 2014-2015 


e Demographic information for 2014-2015 students who were included in the analyses of academic 
performance, including numbers that allow calculation of the results in Figures 4—7 (pages 10-12) 
can be found in Table 3 (page 21). Shown in Figure 4 (page 10), of students who attended 2014 IR 
schools and took both STAAR Mathematics tests, a total of 7,138 students, 52 percent used TTM and 
48 percent did not, while at schools that met the standard, with a total of 50,233 students, a slightly 
smaller percentage, 49 percent, used TTM while 51 percent did not. 
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Figure 4. Percentage of TTM users and non-TTM users among students who took both the 
spring 2014 and 2015 STAAR Mathematics assessments, 
by 2014 school accountability rating 
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Figure 5. Percentage of TTM users and non-TTM users among students who took both the 
spring 2014 and 2015 STAAR Mathematics assessments, by 2014-2015 grade level 
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Grade3 Grade4 Grade5 Grade6 Grade? Grade8 
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Students in grade three took the grade three level STAAR Mathematics test in 
both 2014 and 2015, indicating that they had been retained in third grade. 
Percentages may not equal 100 due to rounding. 

PEIMS Fall Resubmission and TTM files 
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Illustrated in Figure 5 (page 10), in the elementary grades three through five, close to 60 percent of 
students who took both the spring 2014 and 2015 STAAR Mathematics exams were enrolled in TTM, 
while in the middle school grades seven and eight, fewer than 40 percent of students who tested both 
years were enrolled in the program. 


By gender and race/ethnicity, shown in Figure 6, TTM users and non-TTM users were relatively 
equally represented in each grouping. The lowest percentage of TTM users by race/ethnicity was for 
African Americans at 48 percent, and the highest was for American Indian students and students of 
Two or More race/ethnicities, at 53 percent. 


Figure 6. Percentage of TTM users and non-TTM users among students who took both the 
spring 2014 and 2015 STAAR Mathematics assessments, by gender and race/ethnicity 
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Note: Percentages may not equal 100 due to rounding. 
Sources: PEIMS Fall Resubmission and TTM files 


Illustrated in Figure 7 (page 12), when grouped by economic disadvantage, English proficiency, 
special education status, or gifted/talented status, generally half of students who took both the spring 
2014 and 2015 STAAR Mathematics exams used TTM and half did not. The biggest difference in 
percentage between the groups was for students who were not economically disadvantaged (48 
percent of students who were not economically disadvantaged were TTM users and 52 percent were 
non-TTM users). 
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Figure 7. Percentage of TTM users and non-TTM users among students who took both 
the spring 2014 and 2015 STAAR Mathematics assessments, by economic disadvantage, 
English proficiency, special education status, and gifted/talented status 
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Note: Percentages may not equal 100 due to rounding. 
Sources: PEIMS Fall Resubmission and TTM files 


What was the STAAR Mathematics performance of HISD students who used TTM compared with 
the performance of HISD students who did not use the program in 2014-2015? 


e Depicted in Figure 8 (page 13) and detailed in Table 4 (page 22) and Table 5 (page 22), the STAAR 
Mathematics performance of TTM users largely parallelled that of non-TTM users in both 2014 and 
2015. In both years, a slightly lower percentage of TTM users than non-TTM users met the standard 
to pass the exam but did not achieve the advanced standard, and a slightly higher percentage of TTM 
users than non-TTM users met the advanced standard. 


HISD Research and Accountability 12 


Figure 8. Spring 2014 and spring 2015 STAAR Mathematics performance of students 
who took both assessments, by use of TTM 


100 - 
80 - 
$ Ly 
51.8 95.2 
= : 
2 
® 
os 40 
329 
aoe 29.2 29.4 
20 + 
0+ ] 
Did Not Meet | Met, Not Adv. Advanced Did Not Meet | Met, Not Adv. | Advanced 
2014 2015 
@TTM Users @Non-TTM Users 
Note: Percentages may not equal 100 due to rounding. 


Sources: 2014 and 2015 spring STAAR Mathematics and TTM files 


e Shown in Table 6 (page 23), the STAAR Mathematics mean scale scores of TTM and non-TTM users 
differed only slightly in both 2014 and 2015. In both years, the mean scale scores of non-TTM users 
were somewhat higher than those of TTM users, the standard deviations were slightly lower, and the 
ranges were somewhat larger. 


e Significance of the difference between the 2015 STAAR Mathematics scale scores earned by TTM 
and non-TTM users was tested using propensity score nearest neighbor matching. For the analysis, 
spring 2015 STAAR Mathematics scale scores were compared based on matching students on 
variables which included spring 2014 STAAR Mathematics scale scores, 2014—2015 enrollment in a 
2014 IR school, grade level, gender, race/ethnicity, economic disadvantage, LEP status, special 
education status, and gifted/talented status. All 57,371 students in the sample were included in the 
analysis. Shown in Table 7 (page 23), students enrolled in TTM earned an average of less than one 
scale score point less than matched students not enrolled in the program earned. Students who used 
TTM in 2014—2015 had spring 2015 STAAR Mathematics scale scores that did not differ significantly 
from those of matched students who did not enroll in the program. 
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The number of TTM lessons students completed and the number of TTM on-grade level lessons 
students passed (Figure 2, page 8, and Table 2, page 20) did not offer enough advantage to TTM 
users to differentiate their 2015 STAAR Mathematics scale scores from the scale scores of similar 
non-TTM users. 


Table 8 (page 24) lists 2015 STAAR Mathematics results of TTM and non-TTM users divided into 
subgroups based on performance on the 2014 STAAR and also by attendance at a 2014 IR school. 
For each subgroup, 2015 STAAR Mathematics results were similar between TTM users and non-TTM 
users. 


2015 STAAR Mathematics results for subgroups of TTM and non-TTM users based on 2014 STAAR 
Mathematics performance are illustrated in Figure 9 (page 15). The largest difference found, 2.5 
percentage points, was between students who had achieved the advanced standard on the 2014 
STAAR Mathematics exam and who met the standard but did not meet the advanced standard on the 
2015 test. 


Figure 10 (page 15) depicts the same information in Figure 9, performance of 2014-2015 TTM and 
non-TTM users on the 2015 STAAR Mathematics exam by subgroups based on 2014 STAAR 
Mathematics performance, reformatted to form a “heat map” to highlight trends in students’ 
performance. Red indicates students who either failed to meet the passing standard both years or 
who failed to meet the standard in 2015 after passing the mathematics exam in 2014; green indicates 
students who maintained their passing or advanced standard both years; yellow indicates students 
whose performance rating went down, but who still passed the exam; and blue indicates an increase 
in performance level between 2014 and 2015. 


In each subgroup in Figure 10, the majority of students achieved the same performance level in 2015 
that they had achieved on the 2014 STAAR Mathematics exam, at approximately the same 
percentages between TTM and non-TTM users. TTM was associated with slightly higher rates of 
students achieving or maintaining the advanced standard for each subgroup of students, but the 
difference was the smallest, 0.1 percentage point, for students who had not met the passing standard 
on the 2014 assessment. 
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Figure 9. Spring 2015 STAAR Mathematics performance of subgroups of HISD students 
based on spring 2014 STAAR Mathematics performance, by use of TTM 
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Figure 10. “Heat Map” of spring 2015 STAAR Mathematics performance of subgroups of 
HISD students based on spring 2014 STAAR Mathematics performance 
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who still passed the exam; and blue indicates an increase in performance level between 2014 and 2015. 
TTM users’ performance is depicted in the first bar and non-TTM users’ performance is shown in the second 
bar of each pair. 
Percentages may not equal 100 due to rounding. 

Sources: 2014 and 2015 spring STAAR Mathematics and TTM files 
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e Average mean scale scores earned by each subgroup, based on 2014 STAAR Mathematics 
performance or for attendance at a 2014 IR school, are provided in Table 9 (page 25). The mean 
scale score for non-TTM users in each subgroup was higher than the mean scale score for TTM 
users for both years. However, interestingly, the lowest score in the range of 2015 STAAR 
Mathematics scale scores for TTM users was higher than the lowest score in the range for non-TTM 
users in each subgroup. 


e Propensity score nearest neighbor matching was performed for each subgroup with TTM and non- 
TTM users being matched based on spring 2014 STAAR Mathematics scale scores, 2014—2015 
enrollment in a 2014 IR school, grade level, gender, race/ethnicity, economic disadvantage, LEP 
status, special education status, and gifted/talented status. All students in each subgroup were 
included in the analyses. Regardless of the number of TTM lessons students used, the results 
verified that there were no significant differences in 2015 STAAR Mathematics scale scores between 
TTM users and non-TTM users in each of the matched subgroups (Table 10, page 26). 


Discussion 


Like in 2013-2014, the Think Through Math program was relatively widely used in HISD in 2014—2015. 
Though there was a 15 percent drop in the number of TTM users between the years, from 51,863 in 
2013-2014 to 43,997 in 2014—2015, these students included about half of the HISD students who took 
both the STAAR 2014 and 2015 Mathematics assessments. Students in every grade level participated in 
the program, but the vast majority (95 percent in 2013-2014 and 99 percent in 2014—2015) were in 
grades three through eight, the grade levels for which TTM was funded by the Texas Education Agency. 


Unlike in 2013-2014, use of the TTM program in 2014—2015 was not associated with significant gains on 
the STAAR Mathematics exam for students who used TTM. Using propensity score matching, TTM and 
non-TTM users were matched on factors including 2014 STAAR Mathematics scores, attendance at a 
2014 IR school, demographic factors, and participation in programs for English Language Learners, 
special education, and gifted/talented students. The number of TTM lessons students completed and the 
number of on-grade level lessons they passed did not offer enough advantage to TTM users to 
differentiate their 2015 STAAR Mathematics scale scores from the scale scores of similar non-TTM users 
when they were grouped districtwide, by previous year’s performance on the 2014 STAAR Mathematics 
exam, or by attendance at a 2014 IR school. 


The reasons for the change in impact of the TTM program are worth exploring. Since the program had 
been in use in the district since 2012, a variety of people undoubtedly had experience with it. Students 
may have experienced the program’s motivational system of awarding points for clicking in lessons, and 
earned awards like enhancements for their TTM avatars, opportunities to contribute to select charities, 
and class parties. They may have been familiar enough to know how to reap the motivational benefits of 
the program without putting in the thought needed to learn the mathematics in the lessons. 


Teachers and administrators also had opportunities to benefit from direct experience with the program 
and from the experience of their colleagues. With the extensive monitoring required to prevent students 
“gaming” the TTM motivation system, it is conceivable that teachers could have increasingly focused their 
TTM assignments on only very specific objectives or on only students most likely to work independently, 
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providing less general access to the program. On the other hand, with the increasing use of technology 
and the rising popularity of the “flipped classroom,” it is possible that teachers increased their use of 
supplemental online programs like TTM for homework, with the corresponding breadth of time and 
attention students put into the assignments outside the classroom. 


A compelling explanation could be found in data collection and reporting methods in 2014-2015. 
Transfer of student information between HISD and TTM was not finalized before the beginning of the 
school year, though students did begin working with the program early in the academic year. The 
documentation of TTM work for students who began early was lost when the transfer was finalized, and 
the students’ records for the year were compromised so that numbers of discrete lessons they worked 
with could not be calculated. As a result, the number of discrete lessons could not be reliably calculated 
for use in this report. Thus, the lesson counts used in the analyses in this report very likely included 
duplicate lessons. As an extreme example, TTM estimated that there are approximately 40 on-grade- 
level lessons for each grade 3-8. Estimating 40 lessons at each grade, plus 40 for high school lessons 
that are also available, yields a total of approximately 280 discrete lessons available to TTM users in 
grades three through eight. Nine TTM users completed (and passed) more than 280 on-grade-level 
lessons in 2014-2015. One completed 1,087 on-grade-level lessons and passed 1,070 of them. 
Though large numbers of lessons completed and passed could be an indication of time on task, they may 
or may not be an indication of significant achievement in learning mathematics. Students were 
responsive to the rewards for working in TTM and were not limited to the number of times they could 
access, complete, or pass any single lesson. The loss of documentation of TTM students’ utilization of 
the program from the beginning of the academic year and the lack of reliability in determining the breadth 
of mathematical content students addressed through the program could have contributed to finding a lack 
of impact of the TTM program in 2014—2015. 


All of these potential explanations for a lack of impact of TTM in 2014—2015 are subject to modification. 
Students who are “gaming” the system can be held accountable by their teachers monitoring their 
progress, even daily if necessary. Changes in use of the program based on professional experience 
should be an advantage for impact of the program and can be documented to allow further analysis. And 
finally, data collection and sharing methodologies are fixable. 


Though TTM was not shown to impact STAAR Mathematics scale scores for HISD students in 2015, the 
state of Texas, along with HISD, found that the TTM program had a positive impact on 2014 student 
performance in mathematics. Continued use of the program within the district, particularly if it is funded 
by the state, would be helpful in further exploring the conditions under which the program may be used to 
the advantage of HISD students and which students may benefit from the program. Looking forward, 
teachers and administrators who choose to use the program with their students should be encouraged to 
track their adherence to, and variations from, the parameters recommended by the provider. The TTM 
company recommends that students use the program with fidelity, specifically, that they follow an 
assigned “pathway” and complete—and preferably pass—TTM lessons weekly throughout the school 
year, that students receive significant monitoring as they work through the program, and that teacher help 
be readily accessible. Data documenting the use of these parameters, along with reliable data for student 
utilization and progress in the program, could provide opportunities to identify parameters most useful for 
making TTM a productive supplement to mathematics instruction for HISD students. 
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Table 1. Demographic Characteristics of Students Who Used TTM, 2013-2014 and 2014- 


7A0h Es) 
2013-2014 2014-2015 
N % N % 
School Attended 
IR (2014) School 7,058 13.6 5,770 13.1 
Non-IR School 44,805 86.4 38,227 86.9 
Grade Level 
Prekindergarten 5 <0.1 19 <0.1 
Kindergarten 6 <0.1 23 0.1 
Grade 1 6 <0.1 23 0.1 
Grade 2 101 0.2 24 0.1 
Grade 3 11,944 23.0 9,774 22.2 
Grade 4 11,781 22.7 9,673 22.0 
Grade 5 11,407 22.0 8,628 19.6 
Grade 6 5,115 9.9 6,627 15.1 
Grade 7 5,098 9.8 4,702 10.7 
Grade 8 3,676 Bae 4,284 9.7 
Grade 9 2,143 4.1 164 0.4 
Grade 10 352 0.7 22 0.1 
Grade 11 122 0.2 22 0.1 
Grade 12 107 0.2 12 <0.1 
Gender 
Female 25,345 48.9 21,493 48.9 
Male 26,518 51.1 22,504 51.1 
Race/Ethnicity 
African American 13,652 26.3 10,739 24.4 
American Indian 109 0.2 91 0.2 
Asian/Pacific Islander 1,609 3.1 1,504 3.4 
Hispanic 32,448 62.6 27,838 63.3 
White 3,610 7.0 3,422 7.8 
Two or more 435 0.8 403 0.9 
Economic Disadvantage 
Economic Disadvantage 43,257 83.4 33,443 76.0 
No Economic Disadvantage 8,606 16.6 10,554 24.0 
Limited English Proficiency (LEP) 
LEP 20,626 39.8 17,799 40.5 
Not LEP 31,237 60.2 26,198 59.5 
Special Education 
Special Education 3,740 7.2 3,207 7.3 
Not Special Education 48,123 92.8 40,790 92.7 
Gifted/Talented (GT) 
GT 10,055 19.4 8,795 20.0 
Not GT 41,808 80.6 35,202 80.0 
Total 51,863 100.0 43,997 100.0 
Note: Percentages may not total 100 due to rounding 


Sources: PEIMS Fall Resubmission and TTM files, 2013-2014 and 2014-2015 
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Table 2. TTM Achievement of Students Who Used TTM, 2014-2015 


Mean # On- 
Mean # Lessons Grade-Level 
N % Completed Lessons Passed 
School Attended 
IR (2014) School 5,770 13.1 12.8 1.4 
Non-IR School 38,227 86.9 14.6 4.3 
Grade Level 
Prekindergarten 19 <0.1 22.5 1.7 
Kindergarten 23 0.1 28.2 3.3 
Grade 1 23 0.1 10.6 1.5 
Grade 2 24 0.1 27.3 4.5 
Grade 3 9,774 22.2 13.0 6.0 
Grade 4 9,673 22.0 13.7 3.5 
Grade 5 8,628 19.6 13.1 2.9 
Grade 6 6,627 15.1 20.0 4.8 
Grade 7 4,702 10.7 15.8 3.1 
Grade 8 4,284 9.7 11.4 2.1 
Grade 9 164 0.4 8.7 0.8 
Grade 10 22 0.1 16.8 1.4 
Grade 11 22 0.1 13.2 1.5 
Grade 12 12 <0.1 11.8 0.3 
Gender 
Female 21,493 48.9 12.9 3.5 
Male 22,504 51.1 15.8 4.3 
Race/Ethnicity 
African American 10,739 24.4 13.2 2.6 
American Indian 91 0.2 17.7 5.6 
Asian/Pacific Islander 1,504 3.4 31.6 20.3 
Hispanic 27,838 63.3 13.0 2.7 
White 3,422 7.8 20.4 10.7 
Two or more 403 0.9 22.4 10.7 
Economic Disadvantage 
Economic Disadvantage 33,443 76.0 12.9 2.5 
No Economic Disadvantage 10,554 24.0 19.0 8.5 
Limited English Proficiency (LEP) 
LEP 17,799 40.5 13.4 2.6 
Not LEP 26,198 59.5 15.0 4.9 
Special Education 
Special Education 3,207 7.3 13.9 1.0 
Not Special Education 40,790 92.7 14.4 4.2 
Gifted/Talented (GT) 
GT 8,795 20.0 20.0 11.0 
Not GT 35,202 80.0 12.9 2.2 
Total 43,997 100.0 14.4 3.9 
Note: Percentages may not total 100 due to rounding 


Sources: PEIMS Fall Resubmission and TTM files, 2014-2015 
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Table 3. Demographic Characteristics of HISD Students Who Took the Spring 2014 and 


2015 STAAR Mathematics Assessments and Were Used in Analyses, by Use of 


TIM 
TTM Users Non-TTM Users Total 
N % N % N 
School Attended 
IR (2014) School 3,733 13.1 3,405 11.8 7,138 
Non-IR School 24,679 86.9 25,554 88.2 50,233 
Grade Level 
Grade 3 304 1.1 217 0.7 521 
Grade 4 8,504 29.9 5,830 20.1 14,334 
Grade 5 7,555 26.6 5,797 20.0 13,352 
Grade 6 5,561 19.6 5,299 18.3 10,860 
Grade 7 3,822 13.5 6,429 22.2 10,251 
Grade 8 2,666 9.4 5,387 18.6 8,053 
Gender 
Female 14,040 49.4 14,517 50.1 28,557 
Male 14,372 50.6 14,442 49.9 28,814 
Race/Ethnicity 
African American 6,589 23.2 6,999 24.2 13,588 
American Indian 52 0.2 46 0.2 98 
Asian/Pacific Islander 934 3.3 925 3.2 1,859 
Hispanic 18,543 65.3 18,617 64.3 37,160 
White 2,051 7.2 2,156 7.4 4,207 
Two or more 243 0.9 216 0.7 459 
Economic Disadvantage 
Economic Disadvantage 22,016 77.5 22,145 76.5 44,161 
No Economic Disadvantage 6,396 22.5 6,814 23.5 13,210 
Limited English Proficiency (LEP) 
LEP 11,915 41.9 11,728 40.5 23,643 
Not LEP 16,497 58.1 17,231 59.5 33,728 
Special Education 
Special Education 1,025 3.6 1,066 3.7 2,091 
Not Special Education 27,387 96.4 27,893 96.3 55,280 
Gifted/Talented (GT) 
GT 5,893 20.7 5,650 19.5 11,543 
Not GT 22,519 79.3 23,309 80.5 45,828 
Total 28,412 100.0 28,959 100.0 57,371 


Notes: Students in grade three are those who repeated the grade; they were enrolled in 
grade three for both the spring 2014 and 2015 STAAR Mathematics assessments. 
Percentages may not total 100 due to rounding. 

Sources: Spring 2014 STAAR Mathematics, Spring 2015 STAAR Mathematics, 2014-2015 PEIMS 
Fall Resubmission, and 2014—2015 TTM files 
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Table 4. Spring 2014 STAAR Mathematics Results for All HISD Students Who 


Also Took the Spring 2015 STAAR Mathematics Assessment, by Use of 


TIM 
Met the 2014 Phase- 
Did Not Meet the In 1 Standard but Achieved the 
2014 Phase-In 1 not the 2015 2014 Advanced 
Total Standard Advanced Standard Standard 
N N % N N N % 
TTM Users 28,412 9,220 32.5 14,324 50.4 4,868 17.1 
Non-TTM Users 28,959 9,375 32.4 14,995 51.8 4,589 15.8 
Total 57,371 18,595 32.4 29,319 51.1 9,457 16.5 
Notes: Percentages may not equal 100 due to rounding. 


Sources: 2014 STAAR, 2015 STAAR, and TTM files 


Table 5. Spring 2015 STAAR Mathematics Results for All HISD Students Who 


Also Took the Spring 2014 STAAR Mathematics Assessment, by Use of 
TIM 


Did Not Meet the 


Met the 2015 Phase- 
In 1 Standard but 


Achieved the 


2015 Phase-In 1 not the 2015 2015 Advanced 
Total Standard Advanced Standard Standard 
N N % N N N % 

TTM Users 28,412 8,307 29.2 15,693 55.2 4,412 15.5 
Non-TTM Users 28,959 8,508 29.4 16,550 57.1 3,901 13.5 
Total 57,371 16,815 29.3 32,243 56.2 8,313 14.5 

Notes: Percentages may not equal 100 due to rounding. 

Sources: 2014 STAAR, 2015 STAAR, and TTM files 
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Table 6. Spring 2014 and Spring 2015 STAAR Mathematics Mean Scale Scores for All HISD 
Students Who 


a Kole) qm =f) 14) 
Assessments, by Use of TTM 
Spring 2014 STAAR Mathematics 
Scale Scores 


the Spring 2014 and 2015 STAAR Mathematics 


Spring 2015 STAAR Mathematics 
Scale Scores 


Mean SD Range Mean SD Range 
TTM Users 1539.1 158.3 754 — 2138 1585.6 144.5 931 — 2229 
Non-TTM Users 1556.3 155.1 754 — 2181 1600.8 142.6 771 — 2236 
Sources: 2014 STAAR Mathematics, 2015 STAAR Mathematics, and TTM file 


Table 7. Propensity Score Matching Results for All HISD Students Who Took Both 


the Spring 2014 and 2015 STAAR Mathematics Assessments, by Use of 
TTM 


Mean 2015 STAAR 
Mathematics Scale Score 
TTM 
Users Non-TTM Users 
(N=28,412) (N=28,959) Difference S.E. t 
Before Matching 1585.65 1600.80 -15.16 1.20 -12.64* 
Matched 1585.65 1586.42 -0.77 2.49 -0.31 


Note: * indicates p<.001. 
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Table 8. Spring 2015 STAAR Mathematics Results for Subgroups of HISD 


Students Who Also Took the Spring 2014 STAAR Mathematics 
Assessment, by Use of TTM 


Met the 2015 
Phase-In 1 Achieved the 
Did Not Meet the Standard but not 2015 
2015 Phase-In 1 the 2015 Advanced Advanced 
Students Total Standard Standard Standard 
N N % N % N % 
Failed 2014 STAAR 
Mathematics 
TTM Users 9,220 6,034 65.4 3,148 34.1 38 0.4 
Non-TTM Users 9,375 6,185 66.0 3,164 33.7 26 0.3 
Total 18,595 12,219 65.7 6,312 33.9 64 0.3 
Passed but Did Not 
Achieve Advanced on 
2014 STAAR 
Mathematics 
TTM Users 14,324 2,240 15.6 10,746 75.0 1,338 9.3 
Non-TTM Users 14,995 2,299 15.3 11,575 77.2 1,121 7.5 
Total 29,319 4,539 15.5 22,321 76.1 2,459 8.4 
Achieved Advanced 
Standard on 2014 
STAAR Mathematics 
TTM Users 4,868 33 0.7 1,799 37.0 3,306 62.4 
Non-TTM Users 4,589 24 0.5 1,811 39.5 2,754 60.0 
Total 9,457 57 0.6 3,610 38.2 6,060 64.1 
Attended a 2014 IR 
School 
TTM Users 3,733 1,795 48.1 1,766 47.3 172 4.6 
Non-TTM Users 3,405 1,599 47.0 1,685 49.5 121 3.6 
Total 7,138 3,394 47.5 3,451 48.3 293 4.1 
Note: Percentages may not equal 100 due to rounding. 


Sources: 2014 STAAR, 2015 STAAR, and TTM files 
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Table 9. Spring 2014 and Spring 2015 STAAR Mathematics Mean Scale Scores for Subgroups 


of HISD Students Who Took Both the Spring 2014 and 2015 STAAR Mathematics 
Assessments, by Use of TTM 


Spring 2014 STAAR Mathematics Spring 2015 STAAR Mathematics 
Scale Scores Scale Scores 
Students Mean SD Range Mean SD Range 
Failed 2014 STAAR 
Mathematics 
TTM Users 1382.3 88.8 754 — 1572 1472.0 96.5 931 — 2103 
Non-TTM Users 1403.5 90.8 754 — 1572 1490.8 100.5 771-1931 
Passed but Did Not 
Achieve Advanced 
on 2014 STAAR 
Mathematics 
TTM Users 1561.2 78.6 1392-1777 1598.0 103.0 1005 — 2103 
Non-TTM Users 1581.2 79.2 1392-1777 1616.8 105.7 931 — 2103 
Achieved Advanced 
Standard on 2014 
STAAR Mathematics 
TTM Users 1771.2 106.5 1615 — 2138 1764.6 127.1 1359 — 2229 
Non-TTM Users 1787.3 106.3 1615 — 2181 1773.4 126.1 1015 — 2236 
Attended a 2014 IR 
School 
TTM Users 1480.8 143.4 754 — 2064 1529.6 122.0 931 — 1976 
Non-TTM Users 1496.6 139.0 833 — 2138 1550.2 120.3 868 — 2024 


Sources: 2014 STAAR Mathematics, 2015 STAAR Mathematics, and TTM file 
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Table 10. Propensity Score Matching Results for Subgroups of HISD Students Who 


Took Both the Spring 2014 and 2015 STAAR Mathematics Assessments, 
by Use of TTM 


Mean 2015 STAAR 


Students Mathematics Scale Score Difference S.E. t 
Failed 2014 STAAR Wyss Meu Mt 
Mathematics Pea Users 
(N=9,220) (N=9,375) 
Before Matching 1471.97 1490.76 -18.79 1.44 -13.00* 
Matched 1471.97 1471.48 0.49 3.03 0.16 
Passed but Did Not TTM Non-TTM 
Achieve Advanced Users Users 
on 2014 STAAR (N=14,324)  (N=14,995) 
Mathematics 
Before Matching 1598.00 1616.78 -18.78 1.22 -15.40* 
Matched 1598.00 1600.07 -2.07 2.50 -0.83 
Achieved Advanced TTM Non-TTM 
Standard on 2014 Users Users 
STAAR Mathematics | (N=4,868) (N=4,589) 
Before Matching 1764.58 1773.39 -8.81 2.61 -3.38* 
Matched 1764.58 1768.99 -4.41 5.26 -0.84 
TIM Non-TTM 
catia aN a Users Users 
(N=3,733) (N=3,405) 
Before Matching 1529.64 1550.16 -20.52 2.87 -7.14* 
Matched 1529.64 1535.95 -6.31 4.64 -1.36 
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Note: * indicates p<.001. 
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